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ABSTRACT 

This report is intended to provide government 
managers with some perspective on the nature and implications of 
sample surveys, their likely costs, and what can be expected from 
them^ The authors discuss sample surveys less from the technical side 
than from the perspective of the officials who will make management 
decisions about an overall survey. Little attention is devoted to 
specific research methods, though the bibliography lists publications 
that provide such information. Much of the discussion applies to 
surveys in which government employees, rather than private citizens, 
supply the responses. Separate chapters discuss the overall sample 
survey process and the steps involved; the advantages of conducting a 
survey under government auspices, rather than through a university or 
private research organization; the costs involved in each phase of a 
survey; factors affecting the accuracy of a survey; and some further 
considerations in conducting a survey* The appendix contains tables 
that illustrate costs of conducting sample surveys of different types 
and scope* (Author/JG) 
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Summary 



Sample surveys are likely to be extremely useful to state and local 
government managers in the years ahead. Their uses for many purposes seem 
destined to grow rapidly. This report provides guidance to govenunent 
officials who are considering the use of surveys for planning and manage- 
ment. It recommends the basic strategy of sampling so that the needs of 
the total population can be estimated from a relatively small number of 
cases. Surveys can provide indications of citizen attitudes and percep- 
tions and various factual data not readily available through other means. 
Surveys can be a way for local governments to better reflect citizen needs 
and desires. This can be one important aspect of citizen participation* 

Cost is a critical restriction in the use of surveys by local govern- 
ments. This report discusses the cost factors and provides rough estimates 
of the costs of personal interviews (three sample sizes), mail surveys and 
telephone surveys under specific sets of assumptions. The attempt through- 
out is to present "bargain basement" costs. Also, ways are suggested for 
reducing costs through such means as telephone interviews, mail ques- 
tionnaires, volunteer interviewers, contributions of expert services by 
university or civic groups (in the direction, design and analysis of the 
survey program), and the tapping of available federal funds. 

Another major aspect of surveys is the accuracy of the survey results. 
Too often, decisions are made concerning the type of survey and the specific 
information to be sought without considering the likely accuracy of the re- 
sults. Factors affecting survey accuracy are discussed, including the bal- 
ance between cost and accuracy.' 

Suggestions are made for assuring that survey findings will be useful 
to decision makers, for keeping the data confidential, for. avoiding sensi- 
tive questions where possible, for avoiding Implications in the surveys that 
there will be fast improvement in services and thus stirring unrealistic 
citizen expectations, and for dealing with problems in interviewing in low- 
income minority areas that may have been saturated with surveys. 
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I. Introduction 



This report is intended to provide government managers with some per- 
spective on the nature and implications of sample surveys, their likely 
costs, and what can be expected from them. Periodic surveys of citizens' 
perceptions, attitudes, needs, activities, and priorities regarding govern- 
inent services are a potentially powerful tool for both local and state 
governments. They can help determine the adequacy of public services and 
provide guidance on how to improve these services. 

This report is part of an Urban Institute series on "Measuring the 
Effectiveness of Local Government Services." Those completed, on solid 
waste collection and recreation services, and others in preparation give 
instances where information from sample surveys can help governments deter- 
mine how effective their services are. Such surveys can provide both citi- 
zen attitude information, which indicates how effective citizens perceive 
their services to be, and factual data. Exhibit 1 indicates how factual 
data (Measure 1) and opinions (Measure 4) obtained from a sample survey 
might be used. Sample survey data, if properly constructed, can be used 
with other data to indicate how satisfactory a service is. As Exhibit 1 
indicates, survey data can be collected to compare neighborhoods or other 
clientele groups with each other and over time. 

Some examples of information, not readily available through other 
means, that may be obtained from properly formulated sample surveys are: 

--Citizen attitudes toward various types of taxes. 

--Citizen awareness of the availability of various municipal services. 

— Rates of citizen victimization to provide improved information on 
total crime (and not merely reported crime). 

--Citizen feelings of security from crime in various portions of the 
community. 

— Housing conditions. 

— Number of different persons or families using and not using public 
transportation, and their reasons. 

--Transportation delay times and travel times (door-to-door times as 
well as times for each mode). 

--Number of different persons or families using and not using various 
recreational facilities, and their reasons. 

10 
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—Number of sick days and use of medical services and facilities. 

--Citizen perceptions of the adequacy of particular services, such as 
solid waste collection. 

—Citizen perceptions of the courtesy of city employees. 

--Citizen perceptions of the effectiveness of city employees in 
handling complaints. 

Surveys can also be an effective channel for a kind of citizen parti- 
cipation in government. The survey interview gives citizens a chance to 
"talk" to city hall with anonymity. It permits citizens, at least those 
sampled, to make their preferences and displeasures known and in a way that 
can be constructively used by the government. Citizens are likely to wel- 
come the opportunity to voice their concerns, particularly when effective 
action follows the survey. However, if action is not forthcoming, later 
surveys may meet with apathy, if not outright noncooperation. 

SanjEle surveys are generally best for local governments. A complete 
survey of the entire population in a community is too costly and time- 
consuming and, surprisingly, not likely to increase the accuracy of the 
results. Sampling techniques using relatively small numbers of respondents 
are reasonably precise, provided strict procedures are maintained. Complete 
coverage is likely to mean less experienced and less competent interviewers, 
possibly less care in call-backs or location of missing individuals, and 
other "quick and dirty" shortcuts or improvisations. Sample surveys may 
provide information accurate enough for most purposes. Samples of 1000 or 
500 or even less will often provide sufficient precision for many uses. 
Note, for example, that the Gallup and Harris polls normally survey approxi- 
mately 1500 persons to represent nationwide opinion. 

Where relevant, information from surveys can be used in connection with 
data available from other sources. Official records, for example, often 
provide useful information that supplements and amplifies survey data. It 
also becomes possible to cross-analyze sets of data against each other. 

Iflien a time dimension is added, there is even greater pay-off. One of 
the prime purposes of citizen surveys can be to identify changes over time. 
If old issues die and are replaced by new ones, new questions can, of course, 
be added. 

The following discussion looks at sample surveys less from the tech- 
nical side than from the perspective of the officials who will make manage- 
ment decisions about the overall surve/. Little guidance on specific re- 
search methods and techniques is given. Publications on research methodol- 
ogy provide such information; the bibliography gives some suggestions for 
further study. 
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The following sections discuss: (1) the overall sample survey process 
and the steps involved, (2) the advantages of conducting surveys under 
government auspices versus contracting with a university, private research 
organization, or consultant, (3) the costs in each phase of the survey, 
(4) questions of accuracy, and (5) some further considerations* 

Much of the discussion applies to sample surveys in which government 
employees, rather than private citizens, provide the responses (for example, 
inspections by health department medical personnel of health conditions)*^ 

Currently, local governments infrequently use sample surveys, but 
their use is growing. The federal government uses them to obtain national 
health statistics and between-census unemployment and earnings data. At 
the local level, such surveys have played a part in transportation studies, 
in the Model Cities program, and in Community Renewal Program studies. 



^This report deals largely with surveys of citizens, but much of the 
discussion also applies to sampling government records. In practice, much 
valuable but otherwise too costly, and therefore inaccessible, infmnation 
is available by sampling operating records. This can be a very iir rtant 
resource for a government's planning and evaluation activities if the record 
information is reasonably up to date, accurate, and relevant. 

13 
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II. The Survey Process 



Good surveys involve many steps that must be taken in the proper 
sequence. The following list and brief description of basic steps may 
serve as a check list for carrying out the survey process: 

1. Determine the objectives. For what purposes is information needed? 
At least roughly, what degree of accuracy is needed? 

2. Define the population groups to be studied. Are they, for example, 
tenants or homeowners, or both? Workers or residents? Those age 
21 years and over, or 18 and over? 

3. Determine the specific data to be collected and the methods of 
measurement. Should reports of behavior be used or values or 
opinions? Should information on health be obtained by asking 
people or from clinical records? Can data already developed in 
other studies be used or modified for the present purposes? 

4. Choose the sampling unit. Should it be the household, the in- 
dividual, or the city block? What size of sample is needed? 
What should be done about nonresponses? Which sampling method 
is most appropriate and feasible?! 

5. Determine the method of contacting individuals. Should there be 
personal interviews, mailed questionnaires, or telephone calls? 
Should interviews be conducted in specific facilities, such as 
healuh clinics or schools, or at home? 

6. Construct the questionnaire to obtain the desired information. 

7. Organize and carry out the interviews. This includes hiring inter- 
viewers and supervisory, training interviewers, field testing the 
questionnaire and adjusting as necessary, assigning interviews, 
checking on completion and quality of work, collecting and editing 
interview schedules, and dealing with nonresponses and refusals. 



On occasion, it is useful to return to the same respondents in sub- 
sequent surveys. "Panel" surveys may entail added costs in locating the 
same people over a period of time and there are inevitable losses in panel 
members. But these panel studies have advantages, such as identifying 
patterns of activity over time, specifying which kinds of respondents 
change or remain the same, and following up recipients of certain services. 
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8. Process and analyze the data. This includes coding, keypunching, 
tabulating, constructing indices, running cross-tabulations, and 
using appropriate statistical methods to specify relationships 
and significance. 

9. Report the results, indicating implications and recommendations 
for action. 

Conscientious officials will pay close attention to the survey during 
the planning phases. They should attempt to identify as clearly as possible 
what uses they want the information to serve. If decision makers are hazy 
or imprecise about the kind of information they need and from whom they want 
it, the survey director can go far astray. His miscalculations of what the 
decision makers want may not be. caught until the interview schedule is 
printed, the interviewers are in the field, or even until the data are in. 
Correction becomes a costly process* 

Knowing the intended use of the data (1) enables the survey director 
to develop the most appropriate questions and indicators, and (2) gives 
him an idea of how much accuracy is required. Many questions raised by the 
survey director to clarify his tasks may sound like nit-picking. C'Shall 
we ask for information on members of the household who are currently on duty 
in the armed forces?") But they can be important in collecting useful data 
and in comparing data from year to year. The most fruitful procedure is 
for survey staff and officials to engage in give-and-take discussion to 
determine the most practical and useful survey design. 

For some surveys the data may need to be quite accurate, say within 
5 percent. But often wider tolerance is permissible when only large dif- 
ferences in response would affect the action prescribed. Under these cir- 
cumstances, relatively large errors can be tolerated without undermining 
the survey's purposes, and a more economical survey (smaller sample size) 
can be run. 

This, of course, does not imply that sloppy procedures are permissible, 
such as using untrained interviewers or poorly worded questions. Nor should 
it be an excuse for bias — systematic errors that push the data in one direc- 
tion. (For example, a transit analysis should not survey only women who 
are home during the day, thus under -representing low- income and working 
women. ) 



15 



15 



ill. Who Should Do the Survey? 



Local governments frequently are uncertain whether to run surveys 
themselves or to contract them out to more experienced organizations or 
individuals (particularly for the work described in steps 4 through 8 of 
the previous section). Some guidelines may help resolve this choice. 

Advantages of a Permanent In-House Survey Staff 

If surveys are to be conducted regularly, there are advantages in 
developing the capacity to run them within the governmental structure. 

A permanent staff insures continuity of effort from year to year. 
Over the long run, this is likely to keep costs down. 

This core of people, concerned with consistency, can produce findings 
that permit meaningful comparisons over time. 

To the extent the top elective and appointive officials want to en- 
courage their departments or divisions to make greater use of surveys, the 
creation of an in-house staff underscores the seriousness of the government's 
commitment. 

A bjrproduct of a permanent staff is that the talents possessed by sur- 
vey specialists can and should be of considerable assistance to other units 
of the government that deal with the development, collection, analysis and 
use of the data. 



Reasons for Contracting Out Survey Tasks 

Sample surveys need to be conducted by professionally trained personnel. 
Either salary levels or other conditions of local government employment may 
make it difficult to attract capable professionals. 

Survey work tends to be of a periodic nature in most local governments 
with much of the auxiliary staff required only part time. Civil service 
regulations make it difficult in some places to hire the temporary help for 
positions such as interviewers. 

University research groups and commercial survey firms are among the 
organizations whose capabilities and costs may be explored as a viable 
alternative to an in-house survey staff. Cooperative arrangements between 
the government and local college or university can benefit both. 
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Credibility may be an important reason to use an external organiza- 
tion. On sensitive issues where the survey results are to be used as 
guides for public action but where people might assume that the local 
government's own survey would be slanted, it can be advantageous to choose 
a respected and independent survey organization. 

Employing consultants often is advisable for help in organizing and 
carrying out early surveys before the local government develops its own 
proficiency. These consultants, further, may be called on to help recruit 
and develop a permanent staff inside the government which would gradually 
take over the survey operations. 

Final Responsibility 

Whether the technical suirvey work is done by the government's own 
staff, by outside organizations, or by a combination of both, it must be 
stressed that officials within the government must I tar the primary respon- 
sibility. They can neither duck nor delegate the basic decisions on what 
kinds of data are needed, how often, from whom, and with what precisions. 
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IV. Characteristics and Costs 



Cost is a primary factor constraining local governments today from 
using surveys. The major co5ts in sample surveys are listed in Exhibit 2. 

Many factors affect costs, making it difficult to provide "typical*' 
cost estimates. Moreover, prices change from year to year (mostly upward) 
and vary considerably in different locations in the- United States. Govern- 
ment officials should get up^to-date information from a nearby survey 
operation, such as a local health survey, model cities survey, transporta- 
tion study, community renewal program study, health or welfare survey. 
However, as a general guide, some specific estimates of survey costs are 
provided in the Appendix. E^ch main form of interviewing (personal inter- 
view, mail, and telephone) atid their respective costs are discussed here. 
The costs of surveying records are not estimated but would, in general, be 
substantially lower since interviewing, the most expensive element, is un- 
necessary. 



1. The Personal Interview 

Personal interviews ate desirable when (1) the subject matter is com- 
plex, (2) the inquiry is lotig, (3) the survey seeks unstructured, open- 
ended information, or (4) ati5Wers from a sufficient number of respondents 
are not otherwise likely. 

The survey costs of the personal interview are affected by a number 
of factors. Many of these fjtptors also apply to mail and telephone surveys 
and are not repeated in those, discussions. Rough estimates of the costs of 
personal interview surveys. Teased on particular sets of conditions, are in 
Sections A and B of the Appendix. 

a. Experience 

A government that has no prior experience or expertise in sur- 
veys will incur extra starting costs. Expert guidance is needed to avoid 
pitfalls. When the govertim^nt staff gains experience and knowledge, these 
costs should not recur. If periodic surveys are undertaken in order to 
measure trends, the later survey costs will be much less than the initial 
costs. 

b. Availabiljty ^pj an adequate population listing. 

The costs of an acJequate population listing vary considerably, 
depending upon the status of present lists and the work required to make 
them accurate and complete. 
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EXHIBIT 2 

COST ELEMENTS IN SAMPLE SURVEYS BY FUNCTION 

1. Overall survey planning, design, and direction* 

2. Listing the units (blocks, households, individuals, businesses) 
to be sampled. 

3. Selecting 'the sample, preparing maps, instructions, and assign- 
ment schedules. 

4. Developing the interview questionnaire and schedule (including 
pretest costs for interviewing and travel). 

5. Printing questionnaires, training materials, instruction manuals. 

6. Interviewing. 

a. Recruiting and selecting interviewers. 

b. Interviewer training (trainer, training materials, plus 
interviewer time). 

c. Contacting and interviewing, calling back, locating missing 
respondents. 

d. Field supervision. 

e. Travel (time and expenses). 

7. Editing, coding, keypunching, tabulating, computing, and if used, 
computer time. 

8. Telephoning, especially if a survey is by telephone or if inter- 
views are validated by telephone. 

9. Mailing, particularly if a survey is by mail, or when explanations, 
introductions, and appointments are made by mail. 

10. Analyzing data and report writing. 

11. Reproducing the report. 
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In order to draw a sample from a population (or "universe*') , 
the survey director needs a list of that population. Population lists 
may consist of blocks, buildings, housing units, households, and indi- 
viduals. An up-to-date city address directory can be a good source if it 
is complete, accurate, and lists all the dwelling units of the population 
with which the survey is concerned. Other lists— taxpayers, utility users 
voters-- are useful if they do not exclude persons who should be in the 
population to be sampled. Even if they are not complete, they can be used 
to supplement and check other lists. 

If existing lists are out of date, members of the suirvey team 
may have to enumerate dwelling units themselves. Interviewers, with maps 
and assigned territories, might list all dwelling units on each block. 
This means not only homes and apartments, but trailers in trailer camps, 
rooms in lodging houses, living quarters behind stores. At the very least 
if it is not feasible to correct the deficiency, the nature and direction 
of potential bias should be identified and reported. 

Sometimes a survey is concerned not with the total population 
but with some selected segment, such as males 16 to 21 years. If possible 
all people with these characteristics should be listed. Often this is be- 
yond the realm of possibility. A workable substitute is to draw a sample 
from the total universe of households and through "screening** interviews 
to locate the households which contain members with the requisite charac- 
teristics. 

c. The size of the sample . 

Size is probably the most critical single determinant of survey 
costs. Costs increase significantly with sample size. The size of the 
sample depends upon the degree of precision desired. It is necessary to 
consider not only the total sample, but subgroups for which data are to 
be reported (males and females, whites and blacks, persons under and over 
45 years, the major parts of town). The sample sizes for the subgroups 
of concern have to be large enough so that the results can be used with 
confidence. This is discussed at greater length in Section V on "Accu- 
racy." 

d. Type of sample. 

There are different ways of sampling, and their costs vary. 

Simple random sampling involves drawing units at random from the 
whole population. A table of random numbers (which is readily available) 
is traditionally used. 

Stratified random sampling involves drawing samples separately 
from subgroups of the universe (white males, white females, nonwhite males 
nonwhite females). 

Cluster sampling can reduce costs because batches of people who 
live near each other are selected, thus reducing travel time. However, 
for a given sample size, there is somewhat less confidence in the findings 
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In order to reach the best trade-off between cost savings and 
accuracy, an expert's advice is a good investment. 

e . Interviewinj^ . 

Survey organizations by inid-1970 were paying professional inter- 
viewers from $2.00 to $3.00 an hour. Interviewing time includes not only 
actual interviewing hours but time spent in training, pretesting question- 
naires, reading instructions, traveling, locating and contacting respondents, 
recopying scrawled interviews, and turning in completed schedules to the 
office. 

Further costs associated with interviewing are the time and 
money spent for recruiting and selecting the interviewers and for trainers 
and supervisors. Sometimes a validation check is made, a follow-up of a 
sample of respondents by telephone or in person to find out whether they 
were actually interviewed and gave the responses listed. Cheating and 
gross errors are detailed, and the field supervisor eliminates unsatis- 
factory interviewers. 

The length of the interview affects costs. Length is affected 
by the number, type, and complexity of questions, and the number of topics 
covered. In research studies, interviews tend to run one to two hours. 
Government surveys generally take less time. If straightforward data are 
sought and not too many topics are in one interview, 30 to 40 minutes is 
reasonable. However, free-response questions increase time. With free- 
response questions respondents are encouraged to reply in their own words 
rather than giving yes-no, true-false or short answers. For some survey 
purposes it is essential that respondents express themselves fully to avoid 
putting words or interpretations in their mouths. 

Other factors affecting interviewing costs are: 

— Productivity (number of interviews completed per 
hour), depending on the interviewers' ability, 
experience and motivation. 

— Time spent for call-backs to respondents who were 
not at home on the first calls. Procedures should 
limit the number of attempts to reach any one 
respondent (three is a common limit). 

--Neighborhood being surveyed. In some areas, costs 
go up because residents refuse to open their doors 
to strangers, pairs of interviewers are needed for 
security reasons, housing or population listings are 
inadequate, and so forth. 

--Travel time and costs between interviews, and from 
interview locations to the survey office. 



^One study flaunted its eight-hour interview with school superintendents. 

Zl 



ERIC 



21 



— Financial payment to respondents. Most experienced 
survey researchers frown on paying respondents for 
participating, except as a last resort, but some ad- 
vocate it. 

f • Editing, coding, and tabulation of interview questionnaires. 

After completion, the interview questionnaires are checked for in- 
ternal consistencies (''edited'') > verbal answers are converted into numbered 
classifications ("coded"), and the responses typically are put onto punch 
cards. It might seem more economical to tabulate answers by hand if the 
number of interviews is small; however, machine data processing costs are 
not high, particularly if the government already has tabulating equipment. 
Hand tallies use up considerable clerical time. Furthermore, machine pro- 
cessing is almost inevitably faster, more accurate and more economical for 
significant numbers of tabulations (such as for analysis of responses by 
the respondents' age, neighborhood of residence, or by attitude on other 
questions ) • 

Costs relate directly to the sample size and the number and com- 
plexity of questions. Answers that allow easy coding (male/female, age at 
last birthday, approve/disapprove) take little time, whereas long narrative 
responses to ^'experiences I have had with the police" or "suggestions for 
improving health care" require the development of careful classifications 
and then the cross-checking of different coders' decisions on classifica- 
tion. 

After data for more than one year have been collected, the city 
will probably want to compare responses over time. 

g. Analysis and interpretation of results. 

Analytic cost is often overlooked, but unless survey data are 

analyzed, interpreted, and made available to local decision-makers, the 
previous steps are wasted. 

The initial step in analysis is the tabulation of the number of 
people, and the percent of the total, who give each specific answer. 
These findings can then be compared to (1) previous similar surveys, 
(2) surveys on other topics (percentage of people satisfied with recre* 
ational services compared to percentage satisifed with garbage collection), 
and (3) similar surveys in other jurisdictions. 

Further, answers can be broken down by relevant population char- 
acteristics--men compared to women, young to older persons, whites to 
blacks, or newcomers to older residents. Responses tabulated by combina- 
tions of these characteristics (young male newcomers) may provide further 
useful information. 

In addition, answers given on one question can bo rolntod 
answers i^ivon on other quoytions. For exahiplo, it uay ho instructive 
to analyze answers to '-juostioo on satisfaction uiih health sen ices 
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in terms of respondents' replies to questions on the types of health 
services they used. 

One good rule of thumb is to use as variables for analysis those 
factors that can be changed by the government. That is, it is more prac- 
tical to study the effects of factors that can be altered than to look in- 
to the effects of relatively unchangeable attributes. 

On important topics, answers to a number of questions can be com- 
bined into indices (by factor analysis, scaling, etc.) and used for further 
analyses. Without going further into issues of multivariate analysis, it 
can be said that the possibilities (and the costs) of analysis can increase 
almost without limit. If one purpose of a survey is to uncover causal rela- 
tionships (what conditions lead to what attitudes, actions, or conditions), 
sophisticated analysis is vital. More limited analysis will suffice to dis- 
cover the present state of attitudes or the level of current activity. 

For more detailed discussion of the analysis process, several of 
the references in the bibliography provide good discussions. 

2. The Mail Survey. 

The mail questionnaire is the least costly survey method. A rough 
estimate of mail survey costs to a local government, based on one particu- 
lar set of conditions, is in Section C of the Appendix. The mail survey 
can deal with a wide range of questions. If it is well conceived, clearly 
worded, and calls for fixed-alternative (multiple choice) answers that are 
appropriate and easy to check, even people with little education can re- 
spond. The 1970 Census relied mainly on mail returns. 

The major problem with mail questionnaires is that not everybody re- 
turns them. Returns range from 5 to 98 percent, depending on the interest 
of the topic, the population queried, and the relevance and length of the 
questionnaire. Compared with the typical response rate of 80 to 90 percent 
for interview surveys, mail response rates for the general population rarely 
exceed 50 percent without exco.nsive follow-up. A low rate of return re- 
quires that data be treateJ with considerable skepticism in terms of their 
representativeness. The nonrespondents may differ from the respondents in 
important but unspecified ways. The results consequently are not represen- 
tative of the population. Some studies have found that those with extreme 
views are more apt to respond while older and less-educated people and 
women are less apt to return questionnaires. (This conflicts with the ex- 
periences of many city halls where verbal complaints are received from 
older women.) 

Returns can be boosted to relatively high levels by intensive efforts 
(such as letter or telephone follow-ups). A recent report showed mail re- 
turns of 70 percent and over when follow-ups were used. Of course these 



Joseph R. Hochstim and Demetrios \. Athanasopoulos, "Personal Follow 
Up in a Mail Survey: Its Contribution and U-s Cost," Public Opinion 
Quarterly, Spring 1970, Vol. 34, pp. 69-81. 
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extra efforts push up costs, while making the respondent group more repre- 
sentative. Hochstim, for example, gives the costs of his 1965 mail study 
as $37,000 for 5,630 initially returned questionnaires (70 percent of the 
sample), or $6.55 per return. A per'^^nal interview follow-up raised his 
return rate to 86 percent. These additional returns cost $13.65 each. It 
is instructive to note that $10,000 went for supervision, training, record 
keeping, and office work, $5,200 for listing of addresses, and $11,300 for 
enumeration of households. The questionnaires and mailing cost only $6,600. 



Obtaining mail addresses for all units to be sampled is another diffi- 
cult and costly task. 

Some investigators have cut mailing costs by sending questionnaires 
with other mailings, such as utility bills. \^ile this cost-saving device 
nas some merit, it also has limitations: (1) the mailing list may not in- 
clude all the people that the survey aims to reach (or to sample) and may 
include others not appropriate for the survey, and, (2) the replies may be 
emotionally colored by the other items in the envelope. Also, mailing 
costs are usually minor compared to those for planning, design, direction 
accurate enumeration, and so forth. ' 

The potential cost savings will tempt governments to survey by mail, 
particularly as part of regular business, such as utility billings. How- 
ever, bpcause of the dangers described, at least a sample of the nonre- 
spondents should be reached to determine whether their views differ as a 
class from those of the respondents. Probably, a sample of at least 50 
personal interviews (either at home or by telephone) should be included. 



3. The Telephone Survey 

The telephone interview has much to recommend it. Compared to the 
personal interview, travel is eliminated and costs reduced. An estimate 
of telephone survey costs, for one particular set of conditions, is in 
Section D of the Appendix. If no one answers, call-backs are possible at 
different times. Respondents tend to be about as willing to talk and as 
candid as in personal interviews. Furthermore, if calls are made from a 
central location, super^/isors can listen in and correct faulty interview- 
ing styles on the spot. 

Since not everybody has a telephone, this type of interview imposes 
sampling limitations that may be inconsistent with the purposes of the 
survey. It is impossible to rely exclusively on survey by telephone, 
particularly if poor people are to be interviewed, for the data will be 
severely biased by exclusion of the most deprived. The telephone is proba- 
bly ruled out Tor studies of welfare clients, for example. 

In some localities, particularly in cities, many people have unlisted 
telephone numbers. A recent investigation shows that, contrary to popular 
belief, these people are primarily middle-income, not hii;h-income , and are 
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disproportionately black. Survey firms have been devising special proce- 
dures, such as random dialing, to reach them.^ Also, some families have 
multiple listings (both home and office, both husband and wife, teen-age 
children) thereby giving them a higher probability of entering the sample. 
The procedures for a telephone survey should take such factors into account. 

No one has yet documented the maximum length of an interview feasible 
by telephone. Without face-to-face contact there would seem to be limits 
to how long the respondent's attention can be sustained. 

4. Cost-Saving Devices. 

Volunteer interviewers can bring considerable savings. Sometimes 
members of civic or women's clubs, college students, or retired persons 
can be recruited. Also, an arrangement with a nearby college to use the 
survey as "field experience" for a course in research methods, or for a 
faculty meiDber*s own research project, may produce willing hands and pos- 
sibly expert supervision as well. 

Sometimes government personnel are used to interview. This is eco- 
nomical, but there should be sufficient time for training and field work, 
there should be no conflicts of interest (public health nurses asking 
residents about city health services), and it should be clear who is re- 
sponsible for their supervision. 

As already indicated, survey costs may be cut by using telephones or 
mail instead of household interviews when the disadvantages of the cheaper 
methods are not critical. 

Governments may be able to fund at least part of the costs of sample 
surveys through federal programs. Among the potential sources are three 
Department of Housing and Urban Development programs--Community Renewal, 
Model Cities, and "701" Comprehensive Planning Assistance programs-- the 
Department of Transportation's highway planning program (perhaps in coop- 
eration with a regional planning body); and the Law Enforcement Assistance 
Administration's planning grants. Local foundations are other possible 
contributors to surveys. 



1 

See, for exan^ple, «-7, Selected Bibliography. 
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V. Accuracy 



Government managers must also be concerned with the accuracy of survey 
information that enters into their program and policy decisions. 

In any survey there are numerous chances for inaccuracy. Some in- 
accuracies relate to human error, and would be present whether the survey 
were conducted by sampling or by total coverage. These may be called non- 
sampling errors. Statistical formulas will not show ahead of time how 
seriously these errors affect the data. Only painstaking care will control 
them. 

When only a portion of the population is surveyed and is used to rep- 
resent a total population, another source of uncertainty arises. But with 
statistical random s.-^mpling, good estimates can be made as to how precise 
the findings are. 

Inaccuracies stemming from the use of samples will be discussed in 
detail, followed by an accounting of nonsampling survey errors. 

Possible Sources of Error from Use of Sampl es 

Total Population Relative to Sample Size . 

^ As the size of the sample approximates the size of the total popula- 
tion, accuracy goes up. In most practical situations, however, the sample 
size is small relative to the total population. In these cases, the size 
of the population does not significantly affect the statistical precision 
of the sample. The absolute number in the sample is the dominant factor. 

tuldf; rlnn ""^^ population of the United states can run 

under 2,000. (As noted earlier, the Gallup and Harris polls survey about 
J.,5UU persons to represent nationwide opinion.) 



2. Size of the Sample'. 

The basic rule of sampling is that each unit (household, individual, 
city block, or school) must have an equal chance of being sampled. ^ 

hn stratified sampling (see Section IV.l.d), different rates of sam- 
pling may be used for different subgroups. However, members within each 
subgroup must have an equal chance of being selected. 
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Exhibits 3 and 4 illustrate the relationship between sample size and 
precision for random samples. ^ Tables such as these are needed to indicate 
to survey users the magnitude of the risk in accepting sampling findings. 
For example, given a sample size of 500, if 20 percent of the respondents 
give a particular answer, the chances are about 95 out of 100 that the 
true percentage in the total population that would give this answer lies 
betv»een 16.5 and 23.5 percent. To put it another way, there is a 5-in-lOO 
chanc?. that the true percentage lies outside this range. Confidence inter- 
vals can also be computed for measures such as age and income, where aver- 
ages, not percentages, are to be estimated. 

Note that once the sample is reasonably large, increasing the sample 
size has only modest impact on precision. Note that even in the most im- 
precise category of answers shown in Exhibit 3 (50 percent responding 
alike) doubling the sample size from 500 to 1,000 reduces the confidence 
limits by only 2.4 percentage points. 

Very small samples may seriously distort results. Increasing the size 
of the sample increases the precision of survey results, but above a cer- 
tain point the increases in precision become increasingly minor. It is sur- 
prising how often relatively small samples provide sufficiently reliable 
estimates for total population parameters fov many, if not most, purposes. 

The size of the sample depends upon the degree of precision that is 
desired, balanced against cost constraints. It is necessary to consider 
not only the total sample, but also the number and likely size of the sub- 
groups for which the data will be reported (males /females, whites/blacks). 
The sample size in each reported subgroup has to be large enough to keep 
the possible error in the estimate within allowable limits. This is dis- 
cussed further in part (3) of this section. 

Statisticians feel more comfortable with the 95 percent confidence 
intervals of Exhibit 3 than with lower ones. However, for many if not 
most local government uses of survey data, 90 percent confidence intervals 
(see Exhibit 4) may be quite appropriate and would perinit smaller (and 
therefore less expensive) samples. For example, a sample of only 400 
would be needed rather than 500 to provide the confidence intervals in 
Exhibit 3, but there would then be 90 rather than 95 chances out of 100 
that the average response will fall within the indicated confidence limits. 

Statisticians can explain the statistics of these confidence levels, 
but which level to seek--90 or 95 percent or some other--in a particular 
situation is primarily a policy question. Government managers should ask 
technicians to indicate the costs of achieving different levels of preci- 
sion or accuracy, and then weigh these two factors to reach a final deci- 
sion on the selection of the appropriate sampling size. A government's 
estimate of the seriousness of the added risks versus the added samplh \ 
costs is implicit in the selection of the appropriate sample size. 



^The term ''accuracy" is used in this report to include both sampli' . 
and nonsampling errors. The term "precision," hov/ever, is used by statis 
ticians to refer to sampling errors only. 
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If the per- 
cent giving 
the same an- 
swer to a 



EXHIBIT 3 

RELATION BETWEEN SAMPLE SIZE AND PRECISION 
IN A SIMPLE RANDOM SAMPLE 
95 Percent Confidence Intervals 

And the Sample Size is: 
50 100 200 400 500 



1000 



Then there is a 95 out of 100 chance that the percent of the 
total population that would respond the same way would fall 



question is>, within these ranges:^ 



2 


0- 5.9 


0- 4.7 


0.1- 3.9 


0.6- 3.4 


0.8- 3.2 


1.1- 2.9 


5 


0-11.1 


0.7- 9.3 


1.9- 8.1 


2.8- 7.2 


3.1- 6.9 


3.6- 6.4 


10 


1.8-18.2 


4.1-15.9 


5.9-14.1 


7.1-12.9 


7.5-12.5 


8.1-11.9 


20 


8.8-31.2 


12.2-27.8 


14.5-25.5 


16.1-23.9 


16.5-23.5 


17.5-22.5 


50 


36.1-63.9 


40.2-59.8 


43.1-56.9 


45.1-54.9 


45.7-54.3 


46.9-53.1 



NOTES: 1. These apply if simple random sampling is used. 

pling is used, the errors will be greater. 



If cluster sam- 



2. Nonsampling errors are not included. 

3. These apply if the total population from which the sample is 
drawn is large relative to the sample size. If not, the accu- 
racy of the sample estimate should be greater and therefore the 
ranges narrower than given in this table. 

Source: Based on Reference 15 in the bibliography, p. 20 "Standard Error 
of an Estimate of a Proportion in Simple Random Sampling." 



^More precisely, if an infinite number of samples of indicated size 
were taken, 95 percent of them would contain the true value of the total 
population in the given confidence ranges. 
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EXHIBIT 4 

RELATION BETWEEN SAMPLE SIZE AND PRECISION 
IN A SIMPLE RANDOM SAMPLE 
90 Percent Confidence Intervals 

And the Sample Size is: 

If the per- 50 100 200 400 500 1000 

cent giving 

the same an- Then there is a 90 out of 100 chance that the percent of the 

swer to a total population that would respond the same way would fall 

question is, . within these ranges,^ 



2 


0- 5.3 


0- 4.3 


0.4- 3.6 


0.8- 3.2 


1.0- 


3. 


0 


1.3- 3.7 


5 


0-10.1 


1.9- 8.1 


2.4- 7.6 


3.2- 6.8 


3.4- 


6. 


6 


3.8- 6.2 


10 


3.1-16.9 


5.0-15.0 


6.5-13.5 


7.5-12.5 


7.9- 


12. 


1 


8.4-11.6 


20 


10.6-29.4 


13.4-26.6 


15.9-24.1 


16.7-23.3 


17.0- 


23. 


0 


17.9-22.1 


50 


38.3-61.7 


41.8-58.2 


44.2-55.8 


45.9-54.1 


46.4- 


53. 


6 


47.4-52.6 



NOTES: 1. These apply if simple random sampling is used. If cluster sam- 
pling is used, the errors will be greater. 

2. Nonsampling errors are not included. 

3. These apply if the total population from which the sample is 
drawn is large relative to the sample size. If not, the accu- 
racy of the sample estimate should be greater and therefore the 
ranges narrower than given in this table. 



Source: Based on Reference 15 in the bibliography, p. 20 "Standard Error 
of an Estimate of a Proportion in Simple Random Samplin?5/' 



^ore precisely, if an infinite number of samples of iiyJicated size 
were taken, 90 percent of them would contain the true value of the total 
population in the given confidence ranges. 
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3. Number of Subgroups for Which Data Are Reported , 

There will be less precision for subgroups than for the total sample. 
Thus, if survey data is wanted that is analyzed separately by sex (male/ 
female), ethnicity (white/black), and educational attainment (less than 
high school/high school or higher), there will be eight subgroups. 

In analyzing subgroup data, either more "give" in the figures must be 
tolerated or a larger sample used. When the proportion of the population 
in one of the subgroups is very small (few white males with less than a 
high school diploma), a total sample of say 500 may yield too few persons 
to allow generalizations about the subgroup population. For example, if 
a certain response runs about 50 percent on a question for both a total 
sample of 500 and a subgroup of 50 respondents, the 95 percent confidence 
interval would be 46 to 54 percent for the total sample but about 36 to 64 
percent for the subgroup (Exhibit 3). 

It is often desirable to compare the responses of two subgroups. 
Suppose that in a survey of unemployment, the percentage of unemployed 
males in the sample is 5 and the percentage of unemployed females is 8. 
It may be important to know whether this is a real difference or whether 
the difference in the figures could have occurred by chance. If the sam- 
ple size is too small, such differences could statistically have occurred 
by chance even when the true unemployment rate for both subgroups was the 
same. Special statistical tables are needed to compare the significance 
of the differences in responses between subgroups. 



Nonsampling Errors 

1. Interference With the Randomness of the Sample. 

Factors that introduce an unknown bias, such as unknowingly increas- 
ing the probability that some persons are chosen over others, will make 
the sample less representative and consequently will introduce errors into 
the findings. 



2. Accuracy of the Listing From Which the Sample Was Drawn. 

The sample will be less representative if the list is not complete 
and up to date. For as long as the 1970 Census figures are up to date, 
local samples should conform to the ratios of age groups, racial charac- 
teristics, income levels, etc., indicated for the relevant census tracts, 
cities, counties or metropolitan areas. 



3. Conceptualization and Wording of Sun/ey Questions . 

Writing a questionnaire or interview schedule might look simple. But 
the questions, wording, order, and even length can have important effects. 
In 1970 the Gallup and Harris polls obtained rather different responses to 
similar questions on Vietnam because of differences in a few words 
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Furthermore, words have different meanings to different groups. Thus 
when asked ii "hard work would help you get ahead/' black women thought of 
"hard work" as jobs like scrubbing floors. Questions must be worded so 
that people of different education and background perceive common meanings. 
The only sure safeguard against misconceptions is careful pretesting to 
reduce ambiguities and misunderstandings--a vital step in every survey. 

\^en results are to be compared from one period to another, the same 
wording of questions should be retained so that responses are comparable. 
Even minor changes in words can shift responses, as opinion polls have 
demonstrated. 

The order in which questions are asked also has its effect. Early 
questions set the frame of reference for later questions and may focus the 
respondent's attention on matters already raised. Thus, if first asked 
about safety on playgrounds, and later about suggestions for improving 
playgrounds, many respondents will think in terms of improving playground 
safety -^and fail to give other suggestions. Early questions provide cues 
to the respondent; he "learns" what, he thinks, the study wants to hear. 
Many people actually tend to give "acceptable" or "expected" answers, re- 
sponding to these cues by tailoring their answers to fit the situation as 
they perceive it. 

Thus, preparing survey questions requires experienced advice and care- 
ful pretesting. 

4. Performance of the Interviewers. 

Interviewers can introduce errors into the interview in many ways. 
They can fail to state the questions as written and substitute their own 
versions. They can misunderstand the answer, or hear it right and mis- 
record it. They can bias the respondents' answers by suggesting the "right" 
response by word, gesture, or facial expression. Meticulous attention must 
be given to training and supervising interviewers to avoid these fairly com- 
mon errors. 



5. Accuracy of Respondents . 

Respondents may be inattentive, misunderstand the questions, or rattle 
off any answer to fob off an unwelcome interviewer. Sometimes they will 
have forgotten the information asked for, or miscalculate the time period 
specified ("number of visits to the doctor in the past year"). On occasion, 
they will withhold sensitive information or "dress it up" to sound more 
socially desirable. 

The interviewer can minimize such errors by (1) encouraging respondents 
to give their own opinions by assuring them that there is no "right answer"; 
(2) maintaining a businesslike approach, rather than trying to establish a 
chummy, social relationship; and (3) motivating the respondent by clearly 
explaining how the data will be used. 

31 

o 

ERIC 



31 



6. Nonresponses . 

Inevitably, interviewers are unable to contact everybody in the sample 
and some people refuse to be interviewed. (In addition, a small number of * 
people will be unsuitable because of illness, deafness, drunkenness lan- 
guage difficulties or other reasons.) If the nonresponses are above, say, 
15 to 25 percent, the representativeness of the responses and their com- 
parability with data from other years can be seriously questioned. The 
conscientious survey team will redouble its efforts to reach not-at-homes 
by making personal or telephone calls on different days or hours, and to 
convert refusals into interviews by special explanations or offering gifts 
or cash payments. If such an effort is beyond existing financial resources, 
at least a small random sample of nonrespondents should be assiduously fol- 
lowed up to determine whether they differ in systematic ways from the re- 
spondent group. If they do differ, the survey results will have limited 
applicability. Either further sampling will be needed or the description 
of results must include necessary cautions about the bias. 

7- Errors in Coding, Keypunching, and Tabulating. 

Clerical errors will creep in. Most will be random and tend to cancel 
each other. But systematic checks will reduce the possibilities of non- 
compensating errors, such as consistent misclassif ications. 



Other Aspects of A ecu racy 

Small differences in percentages that show up in survey results have 
little significance because of sampling and nonsampling errors. For exam- 
ple, if 21 percent of a sample of 500 persons reported satisfaction with a 
particular service in one year, and 19 percent were satisfied the following 
year, the difference may reflect little more than variations in the sample 
or the survey procedures. Exhibits 3 or 4, though not the appropriate 
tables for comparing the two estimates in this example, do suggest a strong 
possibility that the true figure for the total population in both years may 
be roughly the same. Moreover, these tables consider only sampling error 
and none of the other errors which may affect the data drawn in different 
years . 

Accuracy can be increased basically by increasing the size of the sam- 
ple and by improving the competence of the conceptual, technical, and cleri- 
cal work on the survey. The latter is likely to be more important than in 
makinf the sample design more sophisticated. Ii^iile the potential array of 
errors looks imposing, there are ways to reduce almost every one. Even 
within a stringent budget, conscientious thought and work can reduce error 
to levels that make the data sufficiently accurate for most policy and 
management decisions. 
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VI. Further Considerations 



1. Balance of Costs and Accuracy, 

Both survey costs and desired accuracy affect a final decision on the 
size of a survey sampling effort. The kind of information that should be 
assembled to assist in such a decision for a specific survey is shown in 
Exhibit 5. 

Ideally, the "penalties" of receiving information that has a certain 
statistical uncertainty should be compared with the cost burden of obtain- 
ing more precise information. Yet it would be misleading to suggest that 
the penalties and expenditures can easily be estimated. Even more diffi- 
cult is the prediction of nonsampling errors and the costs associated with 
reducing them. 

All that can be urged is that the survey staff do the best they can, 
learning from their own experience and from other survey teams, to estimate 
both cost and accuracy for several alternative survey designs. On the basis 
of such estimates, local government managers must consider how precisely 
they hope to rely on the survey information. Surveys that aim chiefly to 
show general trends should not involve unnecessary expenses for highly re- 
fined data. Of.her surveys that deal with highly sensitive issues should 
not risk avoidable inaccuracy. Only the policy maker or survey information 
user can judge whether cost or accuracy should take precedence in specific 
cases. 

2. Need to Make Survey Findings Useful to Decision Makers. 

The survey data should be presented in a fovm comprehensible to 
decision makers. Considerable care should be devoted to making results 
clear, interesting, persuasive, and relevant. To bridge the gap between 
survey data and future action, the survey staff should discuss the impli- 
cations of their results and (where the data warrant) offer recommendations. 
It. is also important to indicate clearly the bases in the data for these 
recommendations, and what assumptions are being made. 

Survey reports are enhanced by (1) a short summary of the major find- 
ings and recommendations, (2) good, clear presentation, and (3) for those 
who want details, an appendix on the survey methoJs, including the ques- 
tionnaire used and tables detailing the results. 
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EXHIBIT 5 

COMPARISON OF COST AND ACCURACY LEVELS 
FOR DIFFERENT SAMPLE SIZES AND METHODS 
(rough approximations for illustrative purposes) 



COST LEVEL 



Personnel interview 
Sample of 400 
Sample of 500 
Sample of 1,000 

Telephone interview 
Sample of 500 (in- 
cluding 50 in- ^ 
person interviews) 

Mail questionnaire 
2,000 mailed 
1,000 returned 
(su{>plemented by 
50 telephone and/or 
in-person interviews) 



Total 



$ 9,925 
11,325 
19,550 



8,510 



Approximate 
cost per 
response 



$24.80 
22.65 
19.55 



ACCURACY LEVEL 

The total population 
would differ from the 
sample by plus or 
minus the following 
percentage points for 
a confidence of: 

95 percent 90 percent 



± 4.9 
+ 4.3 
+ 3.1 



± ^-1 
+ 3.6 

+ 2.6 



17.00 



+ 4.3 



+ 3.6 



8,475 



8.10 



^osts are the estimated '^moderate" costs in the Appendix and apply 
to the survey assumptions described there. 

^Accuracy levels are the percentage points (+ or -) by which the sam 
pie percentage could differ froni the ''true'' percentage in the population, 
if the reported percentage is about 40 to 60. (They are obtained from 
Exhibits 3 and 4.) Nonsampling errors are not considered here. 

^If all assumptions of randomization have been met. 

^Because of self-selective nature of returns, this difference will 
inevitably be greater than the earlier sampling tables indicate, but how 
serious this bias is, is very difficult to forecast. 
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3 . Confidentiality. 

The government must guarantee that individual replies on any topic are 
never divulged. This should be standard practice even in types of surveys 
(such as those proposed in The Urban Institute series on "Measuring the 
Effectiveness of Local Government Services"; see Introduction) which do 
not appear to deal with highly sensitive material. Any survey may include 
questions that people are reluctant to answer. For example, age, educa- 
tion, and especially income are private matters to many people. If these 
matters are not of great importance, they can be omitted or made less in- 
trusive, for example, by calling for answers in ranges, "40 to 59 years 
old," "$6,000 to $7,999." 

When a survey asks for criticisms of a governmental unit, the inter- 
viewer should make clear that the complainant will not be identified by 
name, address, or any other characteristic. Unless people are assured of 
anonymity, they may hesitate to speak. 

Of course, promising confidentiality is not enough. Names should be 
removed from interview schedules after they have been checked and punch 
cards made carrying code numbers rather than names. Interviewers must 
understand that they are not at liberty to discuss respondents or to iden- 
tify individuals. Data files always should be accessible only to the sur- 
vey staff. If a respondent by some chance should give incriminating evi- 
dence, it should be held in as much confidence as his other interview 
replies. 



4. Creation of Expectations. 

Interviewing people about public services may lead them to believe 
improvements will be forthcoming. This is particularly likely, of course, 
if the survey receives much publicity, and cooperation is sought on the 
grounds that it will help improve public services. Governments should be 
wary of promising more than they can deliver. Survey purposes should be 
explained realistically. Of course, it would be ideal if public officials 
responded after the survey to indicated public desires for change with 
alacrity. This would create the best climate for surveys in the "long 
run." 



5, Problems in Black Ghettos. 

In recent years many low-income black neighborhoods have been sub- 
jected to study after study. Much has been promised on the basis of these 
studies, but relatively little has come to pass. Therefore, in some areas 
black citizens may be less willing than they once were to answer interview 
questions. To overcome this it should be stressed that no single group or 
neighborhood is singled out for study. It is important to inform them that 
they are included in community surveys because they are citizens and their 
opinions are sought with everyone else's. 
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6' "Indigen ous Interviewers" as Bridge-Builders to the Connnunity, 

The use of interviewers similar to the people they interview in race 
or ethnicity, income level rnd neighborhood has become increasingly common 
in recent years. These "indigenous interviewers," it is often believed, 
generate greater trust, are more favorably received, better understand the 
language and mores of respondents, receive more accurate answers, and even 
make local neighborhoods feel that they are participants in the survey 
process rather than the victims of "outside snoops." 

Professional researchers do not agree that all of these benefits— or 
even many of them--actually accrue. Experience and research suggest that 
accuracy is risked if interviewer and respondent are too much alike, or if 
they are too different, \7here racial issues are not the survey topic, it 
has been found that middle-class white interviewers in black neighborhoods 
apparently complete as high a proportion of interviews and receive as can- 
did answers as black interviewers. (However, there is evidence that on 
racially related issues different-race interviewers do not receive as 
honest answers.) 

Matching interviewer and respondent by race or ethnic group also may 
be preferable if violence or barely suppressed antagonisms are prevalent. 
Obviously interviewers who speak the language are needed in foreign-speaking 
neighborhoods. Citizens in poor neighborhoods may see these interviewing 
jobs as desirable white-collar positions. Jobs may be created and useful 
skills learned if local people are hired as interviewers. Moreover, to the 
extent that local people as interviewers gives people in all neighborhoods 
a stake in the survey, the survey and its subsequent use for decision pur- 
poses will prosper. 



7. Political Consequences. 

Many city officials view statistical data and systematic survey in- 
formation as mixed blessings, \n\ile they recognize the value of such in- 
formation for planning and management, they are also mindful of the poli- 
tical consequences of public knowledge of the survey results. For example, 
release of periodic survey results that show increasing percentages of 
citizens dissatisfied with certain public services may harm the incumbent 
administration politically. 

If surveys are conducted capably, their data on public services are 
neutral only in the sense that they are unbiased estimates of citizen facts 
and opinions; the informat ion can be used equally well by whoever has access 
to it. Officials need not be deterred from using useful tools like surveys 
because their opponents can parade the less favorable information before 
the public. Over the long run, and even the short run, release of such 
information works in favor oi rational public action. For example, it can 
provide potent support for needed government actions. Unless there is an 
unusually sensitive local issue (where information is not likely to be 
heard above the din of argiinent) or public officials are concerned only 
with their images and not the public welfare, knowledge is apt to be more 
useful than ignorance. 

36 



39 

Appendix 



The appendix tables and discussion illustrate some cost ranges based 
upon recent survey experience and other cost information that could be 
synthesized. The three cost estimates in Part A for personal interview 
surveys are based on the assumptions listed below about the nature of the 
surveys. These same assumptions, except for the "in-home personal inter- 
view" aspect of "approach," apply also to Parts C and D for mail and 
telephone surveys. 



A. ILLUSTRATIVE PERSONAL INTERVIEW SURVEY COSTS 

Assumptions '^ 

Population of city or county - 200,000. 

Approach - in-the-hoitve personal interview with any adult member of 
the household • 

Household listing - available but not completely up-to-date ♦ 

Interview training - 3 days of classwork and practice interviews 
with paid trainer. 

Questions - 40 questions, 38 requiring short answers or multiple 
choice, 2 open-ended. 

Interview time - average 30 to 40 minutes. 

Survey planning, direction and analysis--by staff on permanent 
city/county payroll (or "volunteers," e.g., from nearby uni- 
versities). If no staff (or volunteers) are available to 
plan, direct, and analyze findings, salary expense for sur- 
vey staff should be written in (and can be prorated across 
the service areas being helped by the survey). 

Elapsed time - if survey staff are experienced and there are no 
bottlenecks or red tape, three to six months from planning 



The same interview can, and probably should, be used to collect 
information on a number of governmental services. Thus, costs can be 
divided among the agencies involved. 
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to report, for a first effort, ten months might be more realistic. 
(If more staff are assigned, elapsed times can be reduced.) 

Costs are at 1971 levels. 

Note that the relaxation of some of these asstimptions, such as the 
use of a professional survey firm to plan, direct, and undertake 
the survey could raise these costs significantly. 
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1. COST ESTIMATES: 400 INTERVIEWS 

High 
(but not 

Low Moderate highest) 

1. Overall planning and direction 

(see ''Assumptions") « « 

2. Preparing listing $ 250 $1,000 $2,500 

3. Selecting sample, preparing maps, 
assigning households to inter- 
Viewers 100 400 1,000 

4. Developing interview questionnaire 
(including conferences and 20 pre- 
test interviews) 200 600 1,000 

5. Printing interview questionnaires, 
training materials, manuals, sup- 
plies 200 550 950 

6. Interviewing costs 

a) Recruitment - 200 500 

b) Training (20 interviewers for 

20 hours, plus trainer's salary) 1,150 1,500 2,000 

c) Interviewing time (including 

travel, call -backs, recopying) 2,000 3,450 5,600 

d) Field supervision 600 850 1,500 

e) Travel expenses 200 300 400 

7. Editing, coding, punching, tabulat- 
ing 350 700 1,100 

8. Telephoning - lOO 200 

(no extra) 

9. Mailing 10 25 100 

10. Analyzing data and report writing 
(see "Assumptions") - 150 500 

11. Reproducing report 25 100 300 
Total $5,085 $9,925 $17,650 
Average cost per interview $12.71 $24.81 $44.13 



Note: Some of the materials (training materials, household list- 
ings, maps), can be used for other surveys, and not all costs need be ! 
charged against the initial survey. * 

_ J 
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2. COST ESTIMATES: 500 INTERVIEWS 



1. Overall planning and direction 
(see "Assumptions") 

2. Preparing listing 

3. Selecting sample, preparing maps, 
assigning households to inter- 
viewers 

4. Developing interview questionnaire 
(including conferences and 20 pre- 
test interviews) 

5. Printing interview questionnaires, 
training materials, manuals, sup- 
plies 

6. Interviewing costs 

a) Recruitment 

b) Training (20 interviewers for 
20 hours, plus trainer's 
salary) 

c) Interviewing time (including 
travel, call-backs, recopying) 

d) Field supervision 

e) Travel expenses 

7. Editing, coding, punching, tabu- 
lating 

8. Telephoning 

I 9. Mailing 

I 

|10. Analyzing data and report writing 
(see "Assumptions") 

11. Reproducing report 
Total 

Average cost per interview 



Low 



$ 250 



100 



200 



250 



1,150 

2,500 
750 
250 



450 



10 



25 

$5,935 
$11.87 



Moderate 

$1,000 

400 

600 

650 
200 

1,500 

4,300 
1,100 
400 

800 
100 
25 

150 
100 
$11,325 
$22.65 



High 
(but not 
highest ) 



$2,500 
1,000 
1,000 

1,100 
500 

2,000 

7,000 
1,900 
500 

1,300 
200 
100 

500 
300 

$19,900 
$39.80 



Note: Some of the materials (training materials, household list- 
ings, maps), can be used for other surveys, and not all costs need be 
charged against the initial survey. 
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3. COST ESTIMATES: 1,000 INTERVIEWS 



High 
(but not 

Low Moderate highest) 



3. 



Overall planning and direction 
(see "Assumptions") 



Preparing listing 


$ 250 


$1,000 


$2,500 


Selecting sample, preparing maps, 
assigning households to inter- 
viewers 


100 


400 


1,000 


Developing interview questionnaire 
(including conference? and 20 pre- 
test interviews) 


200 


600 


1,000 


Printing interview questionnaires, 
training materials, manuals sup- 
plies 


400 


1,000 


1,600 


Interviewing costs 

a) Recruitment 

b) Training (40 interviewers for 

20 hoiirJ5 nlll^ t*fai np-r * q calaw^ 

c) Interviewing time 

d) Field supervision 

e) Travel expenses 


5,000 
1,400 
500 


300 

J , uuu 
8,600 
2,000 
800 


700 

4 , 000 

14,000 
3,500 
1,000 


Editing, coding, punching, tabu- 
lating 


700 


1,200 


2,000 


Telephoning 




200 


400 


Mailing 


20 


50 


200 


Analyzing data and report writing 
(see ^'Assumptions") 




300 


800 


Reproducing of report 


25 


100 


300 


Total 


$10,895 


$19,550 


$33,000 


Average cost per interview 


$10.90 


$19.55 


$33.00 



Note: Some of the materials (training materials, household list- 
ings, maps), can be used for other surveys, and not all costs need be 
charged against the initial survey. 
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B. EFFECTS OF DIFFERENT ASSUMPTIONS ON COST ESTIMATES 

1. Size of City/County V 

a. In a smaller city, such as one with a population of 50,000 

to 100,000, the costs will not be significantly reduced. The size of sam- 
ple needed for any given level of reliability will be very much the same 
as in a much larger city (Section V). The main savings will be in listing 
costs (fewer dwelling units), and travel time (if there are smaller dis- 
tances). (However, a widely spread-out county of 100,000 may not even 
save on travel.) Total costs are not likely to run more than a few hun- 
dred dollars below the estimates in the previous tables. 

In small cities and counties, the salary scales and the service 
costs may be lower, but this is by no means always the case. If a smaller 
city has lower cost levels, it should look at the columns marked "Low" for 
appropriate estimates « 

b. Concomitantly, a larger city need not spend significantly 
more. Enumeration costs and preparation of maps will be more expensive 
(because of the larger area), and travel time and costs are likely to in- 
crease. 

A heterogenous population may cause costs to increase. A large 
city is apt to have more subgroups about whom separate information is 
wanted (race, income level, section of city, length of residence). If 
this is the case, a larger sample size will be needed and interviewing 
costs will rise. 

2. Survey Planning and Direction. 

We have assumed that the government body would not incur added costs 
for survey planning and direction. Accordingly, not included in the illus 
trative cost estimates is much (in some cases, any) money for (1) overall 
planning and direction and (2) analysis of the data and writing the report 
These are obviously critical to the survey, and require high levels of 
skill. We optimistically (probably over-optimistically) assume that some 
talent is available in or to the government (perhaps from staffs of plan- 
ning, budget, or research functions or local universities) and that these 
people can be used part-year or part-time. 

Another possibility is employing a regular survey director whose sal- 
ary is prorated across all the functional areas served by the surveys. He 
may be able to locate within the government supplementary skills (sampling 
questionnaire construction, field management, data processing, statistical 
analysis), and use appropriate people from other departments as part-time 
staff or consultants. 
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As indicated earlier, outside consultants may be needed at the outset 
of a survey operation* if civic-minded experts donate time, costs will be 
minimal, but regular consulting fees will cause cost estimates (at least 
at the beginning) to be higher than we have indicated. 

Outside research firms' estimates of interviewing costs vary consider- 
ably. We recently asked eight large commercial and academic survey organi- 
zations for their rates. Their figures depend heavily on such things as 
interview length, whether arr^ individual or a certain person (such as the 
head of household) in a household is acceptable, number of call-backs to 
not-at-homes, types of neighborhoods, sampling scheme, and so on* Some 
firms refused to estimate without detailed information on the job. In 
general, their figures tended to lie from $30 to $50 per interview, which 
included assistance with the development of the interview schedule, all the 
interviews, and coding and keypunching (but not analyzing) the data* 

Most firms apparently prefer to err on the high side in their first 
estimate of survey costs, so perhaps a short and simple general-population 
interview would cost somewhat less* Also, these are old, established firms. 
Undoubtedly there are younger, more competitive local firms that will be 
less expensive, in all cases, a knowledgeable government official should 
monitor the survey to be sure it adheres to research standards. 
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C. COST ESTIMATES FOR MAIL QUESTIONNAIRE ("MODERATE")' 



!• Overall planning and direction (see "Assumptions") 

2. Preparing listing (see "Assumptions") 

3. Selecting sample, addressing labels 

4. Developing questionnaire (including conferences 
and 30 pretest interviews) 

5. Printing questionnaires, cover letter, return 
envelopes , supplies 

6. Interviewing 

a. Telephone follow-ups to nonreturners (approxi- 
mately 1,200) 

b. Interviews with 50 nonreturners (to check 
whether they differ from questionnaire re- 
turners) 

(1) by telephone (approximately 30) 

(2) in person (approximately 20) 

7. Editing, coding, punching, tabulating about 1,000 
returned questionnaires) 

8. Telephoning 

9. Mailing, including one mail follow-up to non- 
returners (approximately 1,500) 

10. Analyzing data and report writing (see 
"Assumptions") 

11. Reducing report 

Total 

Average cost of the estimated 1,050 responses 



For 2,000 ques- 
tionnaires mailed 



$1,500^ 
750^ 

600 

1,000 

1,200 



300 
350 



1,200 
300 

875 

300 

100 

$8,475 
$8.10 



^This estimate assumes that about 500 of the 2,000 questionnaires are 
returned. Another mailing goes to the 1,500 nonreturners and about 300 
more returns come in. Telephone calls are made to the 1,200 nonreturners 
to urge cooperation and 200 more responses are received by mail. A random 
sample of 50 is chosen from the 1,000 nonreturners and these are inter- 
viewed by telephone or in person. 

^These costs could be reduced if the mailing was part of other govern- 
ment business. But note the limitations of this approach discussed in 
Section 2 of Chapter IV. 
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D. COST ESTIMATES FOR TELEPHONE INTERVIEWS ("MODERATE") 



For 500 tele- 
phone intervievs ^ 

1. Overall planning an(i direction (see 
"Assumptions^O 

2. Preparing listing g 50o 



3. Selecting sample 30o 

4. Developing interview questionnaire (in- 
cluding conferences, 20 pretest personal 
interviews, 10 pretest telephone inter- 
views goo 

5. Reproducing interview questionnaires, 

training materials, manuals, supplies 650 

6. Interviewing costs 

a. Recruitment 200 

b. Training (20 interviewers for 15 

hours, plus salary of trainer) 1,000 

c. Interviewing time 1 900 

d. Supervision 800 

e. Supplementary in-person interviews 
for 50 persons without telephones or 

not answering 1,000 

7. Editing, coding, punching, tabulating 800 

8. Telephoning 500 

9. Mailing iq 

10. Analyzing data and report writing (see 

"Assumptions*') j50 

11. Reproducing report 100 

Total $8,510 
Average cost per interview $17.00 



Supplemented by 50 in-person interviews of persons without phones 
or not answering. 
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